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Abstract. We present two modified versions of the primahdual sphtting algorithm relying 



on forward-backward splitting proposed in 21 for solving monotone inclusion problems. 



Under strong monotonicity assumptions for some of the operators involved we obtain for 
the sequences of iterates that approach the solution orders of convergence of 0{^) and 
0{ljJ^), for Lo G (0,1), respectively. The investigated primal-dual algorithms are fully de- 
composable, in the sense that the operators are processed individually at each iteration. 
We also discuss the modified algorithms in the context of convex optimization problems 
and present numerical experiments in image processing and support vector machines clas- 
sification. 
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1 Introduction and preliminaries 

The problem of finding the zeros of the sum of two (or more) maximally monotone op- 
erators in Hilbert spaces continues to be a very active research field, with applications 
in convex optimization, partial differential equations, signal and image processing, etc. 

21] ) . To the most prominent methods in this area belong the proximal 
point algorithm for finding the zeros of a maximally monotone operator (see [17| ) and the 
Douglas-Rachford splitting algorithm for finding the zeros of the sum of two maximally 
monotone operators (see ^14j). However, also motivated by different applications, the re- 
search community was interested in considering more general problems, in which the sum 
of finitely many operators appear, some of them being composed with linear continuous 
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operators [1||9. 12 . In the last years, even more complex structures were considered, in 
which also parallel sums are involved, see (6|[7| [T3|[2l] . 

The algorithms introduced in the literature for these issues have the remarkable prop- 
erty that the operators involved are evaluated separately in each iteration, either by for- 
ward steps in the case of the single- valued ones (including here the linear continuous oper- 
ators and their adjoints) or by backward steps for the set-valued ones, by using the corre- 
sponding resolvents. More than that they share the common feature to be of primal-dual 
type, meaning that they solve not only the primal inclusion problem, but also its Attouch- 
Thera-type dual. In this context we mention the primal-dual algorithms relying on Tseng's 
forward-backward- forward splitting method (see |9|[T3]), on the forward-backward split- 
ting method (see |21j) and on the Douglas- Rachford splitting method (see [^). A relevant 
task is to adapt these iterative methods in order be able to investigate their convergence, 
namely, to eventually determine convergence rates for the sequences generated by the 
schemes in discussion. This could be important when one is interested in obtaining an 
optimal solution more rapidly than in their initial formulation, which furnish "only" the 
convergence statement. Accelerated versions of the primal-dual algorithm from |13] were 
already provided in jo], whereby the reported numerical experiments emphasize the ad- 
vantages of the first over the original iterative scheme. 

The aim of this paper is to provide modified versions of the algorithm proposed by Yu 
in 21 for which an evaluation of their convergence behaviour is possible. By assuming 
that some of the operators involved are strongly monotone, we are able to obtain for the 
sequences of iterates orders of convergence of ©(i) and 0(0;"), for lo G (0, 1), respectively. 

For the readers convenience we present first some notations which are used throughout 
the paper (see [1}|3|[15||19|[22]). Let ^ be a real Hilbert space with inner product {■,■) 
and associated norm || • || = The symbols ^ and — )■ denote weak and strong 

convergence, respectively. When Q is another Hilbert space and K : T-L ^ Q & linear 
continuous operator, then the norm of K is defined as \\K\\ = sup{||-fCx|| : x £ Ti, \\x\\ < 1}, 
while K* : Q ^ 7i, defined by {K*y, x) = {y, Kx) for all (x, y) £ HxQ, denotes the adjoint 
operator of K. 

For an arbitrary set- valued operator A : V. ^Tiwe denote by Gr ^ = {(x, -u) £ TixTi : 
u £ Ax} its graph, by domA = {x £ V. : Ax 7^ 0} its domain and by A~^ : 7i ^ Ti its 
inverse operator, defined by (n, x) £ Gr A^^ if and only if {x,u) £ Gr A. We say that 
A is monotone if {x — y,u — v) > for all {x,u), {y,v) £ Gr A. A monotone operator 
A is said to be maximally monotone, if there exists no proper monotone extension of the 
graph oi A on H X Ti. The resolvent of A, Ja ■ H ^ %, is defined by Ja = (Id-^ +^4)"^, 
where Id-^ : % — )• ^,Id-^(x) = x for all x £'H,\s the identity operator on H. Moreover, 
if A is maximally monotone, then Ja ■ T-L ^ Ti is single-valued and maximally monotone 
(cf. [l| Proposition 23.7 and Corollary 23.10]). For an arbitrary 7 > we have (see [l| 
Proposition 23.2]) 

p £ J-yAX if and only if [p, 7~"^(x — p)) G Gr A 
and (see [!} Proposition 23.18]) 

^7^1 + T'A'-iA-i o 7""^ Id^ = Id^ • (1) 

Let 7 > be arbitrary. We say that A is '^-strongly monotone if {x—y, u—v) > "y\\x—y\\'^ 
for all {x,u), {y,v) £ Gr A. A single-valued operator A : H ^ V. is said to be ^-cocoercive 
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if {x — y, Ax — Ay) > j\\Ax — Ay\\'^ for all (x, y) £ TixTi. Moreover, A is 'y-Lipschitzian if 
\\Ax — Ay\\ < 7||x — y\\ for all (x, y) £ H x 7i. A single-valued linear operator A : T-l ^ H 
is said to be skew, if {x, Ax) = for sd\ x G Ti. Finally, the parallel sum of two operators 
A, B-.n^n is defined by ADB -.n^n, ADB = {A-^ + B-^)-^. 

The following problem represents the starting point of our investigations (see |21|). 



Problem 1 Let ?^ be a real Hilbert space, zGT-L, A: T-L^Tia maximally monotone 
operator and C : H ^ Ti an r^-cocoercive operator for r] > 0. Let m be a strictly positive 
integer and for any i E {!,..., m} let Gi be a real Hilbert space, ri G Qi, Bi : Qi ^ Qi 
a maximally monotone operator, Di : Qi ^ Qi a maximally monotone and i^j-strongly 
monotone operator for > and Li : % ^ Qi a nonzero linear continuous operator. The 
problem is to solve the primal inclusion 

m 

find X e -H such that z & Ax + ^L* {{BiUDi){LiX - n)) + Cx, (2) 

1=1 

together with the dual inclusion 



find v,^g,, e Gm such that 3x e H : { J^e'dD^kIx- t,U = 1, m. 

(3) 

We say that {x,vi, ■.■,Vm) Hx Qi x ... x is a primal-dual solution to Problem [l| 



if 

z — / L*Vi ^ Ax + Cx and Vi ^ {BinDi){LiX — ri),i = 1, ...,171. (4) 



m 



=1 



If X S is a solution to then there exists {vi,...,Vm) £ ^i x ... x such that 
{x,vi, ...,Vm) is a primal-dual solution to Problem [l] and, if {vi, .■.,Vm) £ ^i x ... x Qm is a 
solution to O), than there exists x ^ H such that {x,vi, ...,Vm) is a primal-dual solution 
to Problem [Tj Moreover, if {x,vi, ...,Vm) £ Tix Qi x ... x Qm is a primal-dual solution to 
Problem [l| then x is a solution to Q and (wi, ...,Vm) £ Qi x ... x is a solution to Q. 

By employing the classical forward-backward algorithm (see [T2||20| ) in a renormed 
product space, Vii proposed in \21\ an iterative scheme for solving a slightly modified 
version of Problem [T] formulated in the presence of some given weights Wi G (0,1], z = 
1, ...,m, with X^i^i Wi = 1 for the terms occurring in the second summand of the primal 
inclusion problem. The following result is an adaption of (l21j Theorem 3.1] to Problem [T] 
in the error- free case and when A„ = 1 for any n > 0. 

Theorem 2 In Problem^ suppose that 

z G ran + ^ L* ((5,nA)(ii • -Vi)) + . 
Let T and Ui, i = 1, ...,m, he strictly positive numbers such that 



2 • min{T ^,a^ \ ■ min{7y,i/i, ...,Vm] 1 



i=l 
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Let (xo, wi,Oj ■■■■.Vmfl) S Qi x ... x and for all n > set: 

Xn+l = JtA [Xn-T[ YT=l ^'"i,n + CXn - z)] 
Un — X-n 

Vi^n+1 = J .o-iK,„ + CFi{Liyn - D'^Vi^n " ri)],i = 1, ...,m. 

Then there exists a primal-dual solution (x, ui, ...,Vm) to Problem^such that Xn^x and 
) {vi, ...,Vm) as +00. 

The structure of the paper is as follows. In the next section we propose under appropriate 
strong monotonicity assumptions two modified versions of the above algorithm which en- 
sure for the sequences of iterates orders of convergence of C'(^) and for w G (0, 1), 
respectively. In Section [3] we show how to particularize the general results in the context 
of nondifferentiable convex optimization problems, where some of the functions occurring 
in the objective are strongly convex. In the last section we present some numerical exper- 
iments in image denoising and support vector machines classification and emphasize also 
the practical advantages of the modified iterative schemes over the initial one provided in 
Theorem [2j 



2 Two modified primal-dual algorithms 

In this section we propose in two different settings modified versions of the algorithm in 
Theorem [2] and discuss the orders of convergence of the sequences of iterates generated by 
the new schemes. 



2.1 The case A + C is strongly monotone 

For the beginning, we show that in case A + C is strongly monotone one can guarantee 
an order of convergence of 0{^) for the sequence {xn)n>o- To this end, we update in 
each iteration the parameters r and fij, i = l,...,m, and use a modified formula for the 
sequence (yn)n>o- Due to technical reasons, we apply this method in case D^^ is equal to 
zero for i = 1, ...,m, that is Di{0) = Qi and Di{x) = for x 7^ 0. Let us notice that, by 
using the approach proposed in [6| Remark 3.2], one can extend the statement of Theorem 
[7] below, which is the main result of this subsection, to the primal-dual pair of monotone 
inclusions as stated in Problem [TJ 

More precisely, the problem we consider throughout this subsection is as follows. 

Problem 3 Let Ti he a real Hilbert space, z€z7i, A: Ti^T-La maximally monotone 
operator and C :% ^ % a monotone and r/-Lipschitzian operator for ry > 0. Let m be a 
strictly positive integer and for any i G {1, ...,m} let Qi be a real Hilbert space, G Qi^ 
Bi : Qi ^ Qi a maximally monotone operator and Li : % ^ Qi a nonzero linear continuous 
operator. The problem is to solve the primal inclusion 

m 

find X such that z G Ax + ^ L*{Bi{LiX - ri)) + Cx, (5) 

i=l 

together with the dual inclusion 

find vi G Qi, ...,Vm G Qm such that 3x G ^ : 11 ^^*7r '^^'^^ ^-.^^ "^.^^ (6) 

[ Vi £ Bi[LiX - ri),z = 1, m. 
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As for Problem [T| we say that {x,vi, ■■.,Vm) G "Hx Qi x ... x is a primal-dual solution 
to Problem [3j if 

m 

z — L*Vi G Ax + Cx and Vi G Bi{LiX — ri),i = 1, m. (7) 

i=l 

Remark 4 One can notice that, in comparison to Problem [T| we relax in Problem [3] the 
assumptions made on the operator C. It is obvious that, if C is a r/-cocoercive operator 
for r/ > 0, then C is monotone and l/r/-Lipschitzian. Although in case C is the gradient 
of a convex and differentiable function, due to the celebrated Baillon-Haddad Theorem 
(see, for instance, [l] Corollary 8.16]), the two classes of operators coincide, in general 
the second one is larger. Indeed, nonzero linear, skew and Lipschitzian operators are not 
cocoercive. For example, when T-L and G are real Hilbert spaces and L :% ^ Q \s nonzero 
linear continuous, then (x, v) i— )• {L*v, —Lx) is an operator having all these properties. 
This operator appears in a natural way when considering primal-dual monotone inclusion 
problems as done in 

Under the assumption that ^ + C is 7-strongly monotone for 7 > we propose the 
following modification of the iterative scheme in Theorem [2} 

Algorithm 5 

Initialization; Choose tq > 0, fij^o > 0, i = l,..,m, such that 

TO < 27/??, A > r/ + 1, TO YZi ^^oll^if < Vl + ro(27 - 7]t^)/\ 
and (xo,wi,o, •••,^^m,o) G "Hx Qi x...x g^- 
For n > set; X„,+i = J( r„/A)A[a^n - (^n/A)( Ya=i ^i^i,n + Cxn - z)] 
On = l/\/l + T„(27-?7r„)/A 

Un — Xn+1 ~\~ Sn{Xn+l Xn) 

Vi,n+1 = J„ R-il^'j.n + (^i,n{Liyn " ri)], 1 = 1, ...,771 
Tn+l = On'Tn, <^i,n+l = Crj^„/0„+i, 1 = 1, ...,m. 

Remark 6 Notice that the assumption tq o"i^o||-^^j|P < ^1 + to(27 — r]To)/\ in Al- 
gorithm [s] is equivalent to ti ^^^^ cjj^oll-^^ilP ^ li being fulfilled if tq > is chosen such 
that 

7/A + VtVA^ + (Erii 'y^fl\\L^\?? + 



To 



< 



Theorem 7 Suppose that A+C is ^-strongly monotone for > and let {x,vi, ...,Vm) be 
a primal-dual solution to Problem^ Then the sequences generated by Algorithm^ fulfill 
for any n > 

\ II —112/ ™ \ ™ II — 112 
-2 +\l-n2^ai,o\\Li\\ 2^ < 

^n+l V i=l ) i=l ^^"^^'O 

\ II — Ii2 ™ II — Il2 II Ii2 n ™ 

2 ^ / . ^ 2 \ / - Xo),Vifl - Vi). 

Moreover, lim nr^ = ^, hence one obtains for {xn)n>o an order of convergence ofO{^). 
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Proof. The idea of the proof rehes on showing that the following Fejer-type inequality 
is true for any n > 



\ II — Il2 II ||2 
A II -||2 , Sr^ \\Vi,n+l — Vi\\ \\Xn+2 — Xn+l\\ 
-o \\Xn+2-x\\ +} \ 2 



2 

y^{Li{Xn+2 - Xn+l), -Wj.n+l + Vi) < (8) 

\ m II _ ||2 II ||2 

II — 1|2 I ll^j," ~ I ll^n+1 ~ Xn\\ 

2 

— y'(Lj(Xn+l - X„), --Uj,™ + Vi). 
1=1 

To this end we use first that in the light of the definition of the resolvents it holds for 
any n > 

A / ™ A 

{Xn+l - Xn+2) - I ^ L*Vi^n+l + CXn+1 " ^ ) + CXn+2 G + C)x„+2. (9) 



Tn+1 

Since A + C is 7-strongly monotone, Q and ^ yield for any n > 

7||Xn+2 - X||^ < ( X„+2 - X, - X„+2) ) + 

\ Tn+1 I 

I / m \ / m 

i=l / \ i=l 



X„+2 - - ( ^ L*Vi,n+l + C'Xn+i " ^ ) + C'x„+2 - ^ Z - L 

A 



(x„+2 - X, Xn+l - Xn+2) + (x„+2 - X, CXn+2 " CXn+l) + 



{Li{Xn+2 - x),Vi- Vi^n+l) 



1=1 

Further, we have 

II ||2 II ||2 II ||2 

/ v Xn-|_i X\\ \\Xn+2 X\\ X^+l Xj2+2 

(x„+2 - X, Xn+l - Xn+2) = (10) 

and, since C is r/-Lipschitzian, 

/ - n \ ^ ^"^"+1 II -i|2 , ^ II ||2 

(x„+2 - X, 6x„+2 - CXn+l) < ||x„+2 - x|| + ||Xn+2 " X„+i || , 

2 2Tn+l 

hence for any n > it yields 
A 

.Tn+1 



+ 27 - r]Tn+l ] \\Xn+2 " xf < 



A A — 

"x„+i - x|p ||Xn+2 - X„+i|P + 2^(Li(x„+2 - x),Vi - Vi^n+l) ■ 



Tn+1 Tn+1 .^^ 



Taking into account that A > r/ + 1 we obtain for any n > that 
A 



Tn+l 

A 



+ 27 - r]Tn+l \\Xn+2 " < 



Tn+1 



\\Xn+l - x\\ 



1 



\Xn+2 - a^n+llP + 2^(Lj(x„+2 " x),Vi - Vi^n+l) ■ (H) 



i=l 



On the other hand, for every i = 1, m and any n > 0, from 
1 



-{Vi,n - Vi^n+l) + Liyn - Vi £ B,- 



(12) 



the monotonicity of ^ and ([T]) we obtain 



< 



1 



1 

1 



{Vi,n - Vi^n+l) + Liyn - Ti - {Ux - Tj), Uj^^+i - 
,n 

{Vi^n - ■Vj,n+1, ■Wi.n+l " Vi) + (Lj(yn " x),Vi^n+l " ^'i) 
1 „ „o 1 „ 



|f^i,n 'I'll 



2ai 



\Vi,n+l - Vi\ 



hence 



I ~ l|2 II — ||2 II ||2 

\'^i,n+l ^ ^i\\ ||^j,n ^«,n+l|| 



(Ti. 



+ 2{Li{yn- x),Vi^n+l-Vi). (13) 



Summing up the inequahties in (11) and (13) we obtain for any n > 

|2 



A 



Tn+l 

A „ 



+ 27 - 7]Tn+l ] \\Xn+2 " + ^ 



< 



Tn+1 



II f'jjn 
1=1 ' 



Vi.n-ViW'^ \\Xn+2 - Xn+lW^ 



Tn+1 



E 

i=l 



^ i n 



(14) 



+2^(Li(x„+2 - yn), -Vi^n+1 + Wj). 

2 = 1 

Further, since y„ = + dn{xn+i — Xn), for every i = 1, m and any n > it holds 

{Li{Xn+2 - yn),-Vi,n+l +Vi) = (Lj(x„+2 - Xn+1 - 6'„(x„+i - X„)) , + Tlj) = 
{Li{Xn+2 - Xn+l), -Vi,n+1 + Wj) - 6n{Li{Xn+l - Xn), -Vi^n + Vi) + 
On{Li{Xn+l - Xn), -Vi^n + Vi,n+l) < 

{Li{Xn+2 - Xn+l), -Vi,n+1 + Vi) - 6n{Li{Xn+l - Xn), -Vi^n + Vi) + 



ll^n+l-Xnll + 



2 ll"^*,?! "^1,71+1 I 
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By combining the last inequality with (14) we obtain for any n > 

\ \ "^11—112 II ||2 

A _ \ II _||2 , \\Vi,n+l — Vi\\ \\Xn+2 — Xn+l\\ 

h 27 - r]Tn+l \\Xn+2 - x\\ + > \ 

Tn+l J ~[ <yi,n Tn+1 

m 

2 y^(Lj(Xn+2 - Xn+l), "t'i.n+l + Vi) < (15) 



i=l 



X \\v- v-ll"^ 



1=1 ' \2 = 1 



After dividing (15) by r„+i and noticing that for any n > 

A 27 A 

— \ V = ^— , 

''"ra+l ''"n+2 
Tn+lO'i^n = ''"raCTi^n-i = ... = TifTj^o 

and 

it follows that the Fejer-type inequality ([s]) is true. 

Let G N, > 2. Summing up the inequality in ([s]) from n = to A^ — 1, it yields 

^ II -||2 , \\'^i,N - ViW^ \\XN+1 - XnW^ . 

+2^ + -2 < 

- + V + + (16) 

™ / 1 1 \ 

2^^ — (Li(xAr+i - XAr),-'Uj,Ar + tJi) - xq), -Ui^o + ^^j) . 

Further, for every i = 1, ...,m we use the inequality 
2 

{Li{xN+l - Xn), -Vi^N + Vi) < 

TN 

2 /^^^m „ |FjV+i - XAr|| H Ht'j.TV - || 

'rN{l^i=l^im\H\ ) cri,0 

and obtain finally 

A||xAr+l - X|p _^ Wvj^N - Vi\? ^ A||X1 - X|p _^ ||fi,0 - _^ ||xi - XqP 



„ m m ii r 112 

H — > {Li{xi - xo),Wj,o , — \\vi,N - ViW , 

1=1 1=1 ' 
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which rapidly yields the inequality in the statement of the theorem. 

We close the proof by showing that lim nr„ = A/7. Notice that for any n > 0, 

n—>-+oo 

Tn+l = =. (17) 

Since < tq < 27/r/, it follows by induction that < Tn+i < Tn < tq < 2j/r] for any 
n > 1, hence the sequence (r„)„>o converges. In the light of (17) one easily obtains that 
lim Tn = and, further, that lim = 1. As (— )n>o is a strictly increasing and 

n-s>+oo n-s>+oo "^"+1 "^^ - 

unbounded sequence, by applying the Stolz-Cesaro Theorem it yields 

n T n+l-n T„r„,+i 



lim riTn = lim -j- = lim — 5 ^- = lim 

n— ^+00 n— >+oo — 71— >+oo — i — — — n— >+oo T-n — ''"n+1 

(^n + Tn+l) 



hoo 



-2 - t2^i n^+00 r2^^ ^ (27 - r]Tn) 



y Tn + Tn+i r„+i +1 A 

iim = lim — ■ = — 

Tn+U ;^ ^TnJ ^ ^Tn 1 



Remark 8 Let us mention that, if ^ + C is 7-strongly monotone with 7 > 0, then the 
operator A + X^i^i ^*iXBi{Li ■ —ri)) + C is strongly monotone, as well, thus the monotone 
inclusion problem ([5| has at most one solution. Hence, if (x, ui, I'm) is a primal-dual 
solution to Problem |3j then x is the unique solution to ([s]). Notice that the problem ^ 
may not have an unique solution. 

2.2 The case A + C and B^^^ + D^^,i = 1, ...,m, are strongly monotone 

In this subsection we propose a modified version of the algorithm in Theorem [2] which 
guarantees when A + C and + D~^,i = l,...,m, are strongly monotone orders of 
convergence of 0{uj'^), for u G (0, 1), for the sequences (2;n)n>o and {vi^n)n>o,i = 1, ■■■,m. 
The algorithm aims to solve the primal-dual pair of monotone inclusions stated in Problem 
[1] under relaxed assumptions for the operators C and D^^,i = 1, ...,m. 

Problem 9 Let ?^ be a real Hilbert space, z^Ti, A: T-L^Tia maximally monotone 
operator and C : % ^ % a, monotone and ry-Lipschitzian operator for r/ > 0. Let m 
be a strictly positive integer and for any i E {l,...,m} let Qi be a real Hilbert space, 
'Ti ^ Qi, Bi : Qi ^ Qi a maximally monotone operator, Di : Qi ^ Qi a monotone operators 
such that is i/j-Lipschitzian for > and Li : % ^ Qi a nonzero linear continuous 
operator. The problem is to solve the primal inclusion 

m 

find X G ^ such that z e Ax + ^L* {{BiUDi){LiX - n)) + Cx, (18) 

i=l 

together with the dual inclusion 

find e gi, ...,vn. e Gm such that 3xen: | 1~J(b=^^d^^11^1 ^^^^-^ 1^ 

(19) 
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Under the assumption that A + C is 7-strongly monotone for 7 > and B^^ + D^^ 
is (5j-strongIy monotone with 6i > 0,i = 1, ....m, we propose the following modification of 
the iterative scheme in Theorem [21 

Algorithm 10 

Initialization; Choose fj, > such that 

^^ < min {tV^', Sf/uf, ...,6l/ul, ^/^7TS^PTO)}, 
T = n/ (27), ai = /i/ {26i), i = 1, .., m, 

e G [2/(2 + ^), 1] and (xq, fi,o, ?^m,o) G T^x x...x Qm- 
For n > set; Xn+i = JtA [xn - t[Ya!=i L*i'"i,n + Cxn - z)] 

Vn = Xn+l + 0{Xn+l - Xn) 

Vi,n+1 = J„-B-^['"hn + CniLiVn - D^^Vi^n - Ti)], i = 1, ...,m. 

^ i 

Theorem 11 Suppose that A + C is ^-strongly monotone for 7 > 0, B^^ + D^^ is 6i- 
strongly monotone for 6i > 0, i = 1, ...,m, and let {x,vi, ...,Vm) be a primal-dual solution 
to Problem^ Then the sequences generated by Algorithm^^ fulfill for any n > 



7\\X: 



i+l - x|p + (1 - Uj)'^di\\vi^n - ViW'^ < 



i=l 

I 7l|a;i - x\\^ + ^^5i\\vifl - Vi\\^ + - xo|P + /i(iJ^(Li(xi - xo),fi,o - Vi) 

\ i=l i=l 

where < u = < 1. 



Proof. For any n > we have 



^{Xn+l - Xn+2) - i L*Vi^n+l + CXn+l - z] + Cx„+2 ^ {A + C)Xn+2, (20) 



thus, since ^ + C is 7-strongly monotone, (19) yields 

l\\Xn+2 - x\\'^ < (xn+2 - X, - Xn+2) ) + 



m 



Xn+2 - X, 



L*Vi,n+l + CXn+l - + CXn+2 - [^-^Z ^i^'' 



\ \i=l / \ i=l 

1 



r 



{Xn+2 - X, Xn+l - Xn+2) + {Xn+2 - X, CXn+2 - CXn+l) + 



{Li{Xn+2 - x),Vi- Vi^n+l) 



1=1 



Further, by using ( 10 ) and 

2 

{Xn+2 -X,CXn+2 " CXn+l) < 7;\\Xn+2 " + — \\Xn+2 " Xn+lW"^ , 

z 27 
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we get for any n > 



1 7\ „ -„2 
- + -J||x.+2-x|| < 

1 „ -„2 ( 1 



)m 
\\Xn+2 - Xn+l\\'^ + ^(L^(x„^ 
i=l 



After multiplying this inequality with /U and taking into account that 
we obtain for any n > 

l{l + ^]\\Xn+2-x\? < (21) 



2 



7||x„+i - x|p - -||x„+2 - Xn+ill^ + /i^(Li(x„+2 - - i;i,„+i). 

i=l 

On the other hand, for every i = 1, m and any n > 0, from 

— (I'i.n - Vi^n+l) + LiVn " D'^Vi^n " + Dj'^t'i^n+i G (5^^ + Dr^)?;. (22) 

the 5j-strong monotonicity of B^^ + and (19) we obtain 

Si\\Vi^n+l -ViW'^ < ( — {Vi^n - Vi,n+l),Vi,n+l -Vi 

\cri 

+ {Liyn -ri- D'^Vi^n + Dr^Vi^n+i - {LiX - ri),Vi^n+i - Vi) . 
Further, for every i = 1, m and any n > we have 

and, since D^^ is a i/j-Lipschitzian operator, 

Consequently, for every i = 1, m and any n > it holds 

1 u - 1,2 
7^\\Vi,n - - ( TT ^ ) ll^'i.n+l " '^i,n\\'^ + {Uix - Vn) -.Vi - l'j,„+l), 

which, after multiplying it by /i (here is the initial choice of fi determinant), yields 

s 

Si + Wvi^n+l-ViW"^ < Si\\Vi^n-Vif-^\\Vi,n+l-Vi^nf+fi{Li{x-yn),Vi-Vi^n^^ (23) 

11 



We denote 

m 

an ■■= 7lkn+l - ^11^ + ^^ill^i," - "^iW"^ ^'^ > 0- 
1=1 



Summing up the inequalities in (21) and (23) we obtain for any n > 

1 + ^) an+i < an (24) 



m ^ m 

-^\\Xn+2 - a^n+llP - ^ -^\\Vi,n " ""^n+llP + fJ-^{Li{Xn+2 " yn),Vi - Wj,n+l)- 
1=1 i=l 

Further, since y„ = Xn+i + 9{xn+i — Xn) and oj < 6, for every i = 1, m and any n > 
it holds 

{Li{Xn+2 - yn),Vi - Vi,n+l) = (-^^i [Xn+2 " a:n+l - 6'(x„+i - Xn)) ,Vi - t'i,n+l) = 
{Li{Xn+2 - Xn+l),Vi - Vi^n+l) - Uj{Li{Xn+l - Xn),Vi - Vi^n) + 
L0{Li{Xn+l - Xn), Vj,n+1 - Vi^n) + {0 - u){Li{Xn+l - X„), Uj,„+l - Vi) < 
{Li{Xn+2 - Xn+l),Vi - Vi^n+l) " w(Lj(x„+i - Xn),Vi - Vi^n) + 



25j 2iiuj\ 

{Li{Xn+2 - Xn+l),Vi - Vi^n+l) " w(Lj(x„+i - - Vi^n) + 

II l|2 II ||2 II — ||2 

+ 6. + {e-u)6. — . 

Taking into consideration that 

> -^-^ < — < and 1 + - = - H , 

i=l 



from (24) we obtain for any n > 

1 7.. ..o 

-a.n+1 + - Fn+2 - Xn+l\\ < 

OJ 2 

7 II \\2 ^ ~ ^ I V^'^iii - ii2 1 

an + ^^\\Xn+l - Xn\\ — I a„+i - 2^ — || TJj^n+l - || 1 + 

m 
1=1 

As UJ < 6 and a„_|_i — X^I^i — > 0, we further get after multiplying the last 

inequality with w""" the following Fejer-type inequality that holds for any n > 

m 



i=l 
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Let G N, > 2. Summing up the inequality in (25) from n = to — 1, it yields 



2 

i=l 

Using that 

/ T / \ — \\ ||2 — ||2 - 1 

{Li{XN+l - XN),Vi^N - Vi) > — \\xn+1-Xn\\ \\Vi,N - Vi\\ ,1 = 1,. ..,171, 

this further yields 



-N , -N+l \ 1 Sr^ -^i 1 II ||2 -Af+1 x II - I|2 ^ 

W QN+UJ ^ I ^ - Y / . j I 11^^ ~ -W ^ 2_^di\\Vi^N-Vi\\ < 

\ j=l * / i=l 

m 

ao + - xolP + ;Uiu^(Li(xi - xo),fj,o -Vi). (26) 



i=l 

Taking into account the way /x has been chosen, we have 

2 II r Il2 

7_/^y^^ 7_7 
2 i ^ 5i - 2 4 ' 

i=l 



hence, after multiplying (26) with uj it yields 



OAT - 0J^6i\\Vi^N - ViW'^ < f ao + '^oj\\xi - xo|P + HUJ^{Li{xi - Xq), Vi,0 - Vi) \ . 
1=1 \ i=l ) 

The conclusion follows by taking into account the definition of the sequence (a„)„>o. ■ 

Remark 12 If j4 + C is 7-strongly monotone for 7 > and + is Jj-strongly 
monotone for 5, > 0, i = 1, ...,m, then there exists at most one primal-dual solution to 
Problem^ Hence, if {x,v\, ...,Vm) is a primal-dual solution to Problem^ then x is the 



dual inclusion (19). 



unique solution to the primal inclusion (18) and (vi, ...,Vm) is the unique solution to the 



3 Convex optimization problems 

The aim of this section is to show that the two algorithms proposed in this paper and 
investigated from the point of view of their convergence properties can be employed when 
solving a primal-dual pair of convex optimization problems. 

For a function / : 7^ — >■ M, where M := Mujitoo} is the extended real line, we denote by 
dom/ = {x £ Ti : f{x) < +00} its effective domain and say that / is proper if dom/ 7^ 
and f{x) 7^ —00 for all x £ Ti. We denote by r(^) the family of proper convex and lower 
semi-continuous extended real- valued functions defined on T-L. Let /* : H — s- M, f*{u) = 
sup^jg-^Kn, x) — /(x)} for all u £ T-L,he the conjugate function of /. The subdijferential of 
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fatx£n, with f{x) G M, is the set df{x) := {v £ Ti : f{y) > f{x) + {v,y- x) Vy € V.}. 
We take by convention df{x) := 0, if f{x) G {±00}. Notice that if / S ^(T-l), then df is a 
maximally monotone operator (cf. fio]) and it holds {df)~^ = df* . For f,g:T-L^M. two 
proper functions, we consider their infimal convolution, which is the function fOg : ^ — >• M, 
defined by {fng){x) = infj^e^{/(y) + g{x - y)}, for all x eTi. _ 

Let S CTihe a nonempty set. The indicator function oi S, 5s ■ ^ M, is the function 
which takes the value on 5 and +00 otherwise. The subdifferential of the indicator 
function is the normal cone of S, that is Ns{x) = {u G Ti : {u,y — x) < Vy € S}, if 
X £ S and Ns{x) = for x ^ S. 

When / G ^{'H) and 7 > 0, for every x £ Ti we denote by prox^j(x) the proximal 
point of parameter 7 of / at x, which is the unique optimal solution of the optimization 
problem 

inf + 1 (27) 



yen [ 27 

Notice that J^Qf = {Id-^ +^df)^^ = prox^j, thus prox^j : H ^ T-L is a single-valued 
operator fulfilling the extended Moreau 's decomposition formula 

prox^j +7 proX(i/^)y. 07^1 Id-^ = Id^ . (28) 



Let us also recall that the function / : 7^ — )• M is said to be ^-strongly convex for 7 > 0, 

71 
2 I 



if / — 3 II • IP is a convex function. Let us mention that this property implies 7-strong 



monotonicity of df (see [l] Example 22.3]). 

Finally, we notice that for f = Ss, where S C is a nonempty convex and closed set, 
it holds 

J-yNs = Jns = JdSs = (M^ +Nsy^ = Wo^Ss = ^s, (29) 

where Ps % ^ C denotes the projection operator on 5 (see (1} Example 23.3 and Example 
23.4]). 



In order to investigate the applicability of the algorithm introduced in Subsection 2.1 
we consider the following primal-dual pair of convex optimization problems. 

Problem 13 Let ?^ be a real Hilbert space, z G Ti, f £ ^i'H) and /i : "H — t- M a convex 
and differentiable function with a 7/-Lipschitzian gradient for r] > 0. Let m be a strictly 
positive integer and for any i G {1, m} let Qi be a real Hilbert space, ri G Gi, gi G ^{Gi) 
and Li : % ^ Qi a nonzero linear continuous operator. Consider the convex optimization 
problem 

inf i f{x) + V gi{LiX - n) + h{x) - {x, z) I (30) 
and its Fenchel-type dual problem 



sup 



if*ah*) Z -Y.L*Vi - J2 {9:{V^) + {v^,r^)) • (31) 



Viegi,i=l,...,m ^ y -^-^ J .^-^ 

Considering maximal monotone operators 

A = df, C = V/i and Bi = dgi, i = 1, m. 
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i=l 



the monotone inclusion problem ^ reads 

m 

find X gV. such that z £ df{x) + L*{dgi{LiX — rj)) + V/i(x), 
while the dual inclusion problem ([g]) reads 
find vi G GiT--,Vm G Qm such that 3x € H 
If {x,vi, ...,Vm) S Tix Qi X ... X Q„i is a primal-dual solution to (32)-(33), namely. 

m 

z — L*Vi G df{x) + Vh{x) and S dgi{LiX 



YZlLp^^^f{x) + Vh{x) 



Vi G dgi{LiX 



1 m. 



(32) 



(33) 



, m, 



(34) 



2 = 1 



then X is an optimal solution of the problem (30), {vi, ...,Vm) is an optimal solution of 
(31) and the optimal objective values of the two problems coincide. Notice that (34) is 
nothing else than the system of optimality conditions for the primal-dual pair of convex 
optimization problems (30)-(31). 

In case a qualification condition is fulfilled, these optimality conditions are also neces- 
sary. For the readers convenience, let us present some qualification conditions which are 
suitable in this context. One of the weakest qualification conditions of interior ity- type 
reads (see, for instance, [13| Proposition 4.3, Remark 4.4]) 



(35) 



(ri, ...,rm) G sqri JJdomgj - {{Lix, ...,Lmx) : x G dom/} . 



vt=l 



Here, for T-L a real Hilbert space and S QH a convex set, we denote by 

sqri S := {x G S : U\^qX{S — j;) is a closed linear subspace of Ti} 

its strong quasi-relative interior. Notice that we always have int S Q sqri S (in general 
this inclusion may be strict). If Ti is finite-dimensional, then sqri 5 coincides with hS, 
the relative interior of S, which is the interior of 5 with respect to its affine hull. The 
condition (35) is fulfilled if (i) dom^j = Gi, i = l,...,m or (ii) Ti and Gi are finite- 
dimensional and there exists x G ridom/ such that LjX — G ridom^j, i = l,...,m 
(see 13, Proposition 4.3]). Another useful and easily verifiable qualification condition 
guaranteing the optimality conditions (34) has the following formulation: there exists 
x' G dom / n Pli^i -^r^('^« + dom^j) such that gi is continuous at Ljx' — Vi, i = 1, ...,m 
(see (3| Remark 2.5] and jsj). 

The following two statements are particular instances of Algorithm [5] and Theorem [7j 
respectively. 



Algorithm 14 
Initialization; 



For n > set; 



Choose To > 0, fjj^o > 0, i = l,..,m, such that 

TO < 27/??, A > r? + 1, TO J2T=i ^^fl\\Lif < Vl + ro(27 - rjro)/X 

and {xQ,vifi,...,Vm,o) G Tix x...x g^. 

Xn+l = pTO^(^r„/\)f [^71 " ("Tn/A) ( YaLi L*Vi,n + Vhl^Xn) - z)] 

On = l/Vl + r„(27 - VTn)/X 

Un — ~l~ (^n{Xn+l Xn) 

Vi^n+1 = prOX^^^g. [Vi^n + Cri^niLiVn " Ti)], 1 = 1, m 
Tn+1 = Qn'Tn, <^i,n+l = <^i,n / (^n+l , = 1, ■■■■,m. 



15 



Theorem 15 Suppose that f + h is ^-strongly convex for 7 > and the qualification 
condition (35) holds. Then there exists a unique optimal solution x to (30), an optimal 
solution {vi, ...,Vm) to (31) fulfilling the optimality conditions (34) and such that the op- 
timal objective values of the problems (30) and (31) coincide. The sequences generated by 
Algorithm I4 fulfill for any n > 



Allx. 



71+1 



A||xi 



'n+l 



X\ 



Vi. 



< 



\ i=l / 1=1 

m XX _ ||9 II ||0 

Vjfi - Vi\\ IFi - xqII^ 



+ 



2 



1=1 



1=1 



i{Xi - XQ},Vin - Vi). 



Moreover, lim nr^ = hence one obtains for {xn)n>o order of convergence ofO{-). 

n—^+oo 7 — " 

Remark 16 The uniqueness of the solution of (30) in the above theorem follows from [l| 
Corollary 11.16]. 



Remark 17 In case h{x) = for all x ^ T-L, one has to choose in Algorithm 14 as initial 
points To > 0, (Ti,o > 0, i = l,..,m, with tq ^^^^ (Ti,o||i^i|P < y^l + 2toj/ A and A > 1 
and to update the sequence {On)n>o via On = + 2r„7/A for any n > 0, in order to 

obtain a suitable iterative scheme for solving the pair of primal-dual optimization problems 



(30)-(31) with the same convergence behavior as of Algorithm 14 



We turn now our attention to the algorithm introduced in Subsection 2.2 and consider 
to this end the following primal-dual pair of convex optimization problems. 

Problem 18 Let Tihe a real Hilbert space, z £ H, f ^ TCH) and /i : 7^ — )• M a convex and 
differentiable function with a r/-Lipschitzian gradient for r] > 0. Let m be a strictly positive 
integer and for any i G {1, m} let Qi be a real Hilbert space, G Qi, gi, li G T{Oi) such 
that li is f~ ^-strongly convex for i/j > and Li : Ti ^ Qi a nonzero linear continuous 
operator. Consider the convex optimization problem 



inf I f{x) + 'S^{giUli){LiX - ri) + h{x) - (x, z) 
x&'H I ^ — ' 



(36) 



i=l 



and its Fenchel-type dual problem 



sup -(/*□/!*) Z -J2L*^V^ - idKv^) + + {v^,r^)) • (37) 

Vi(^g,,z=l,...,m [ \ .^^ J .^^ J 

Considering the maximal monotone operators 

A = df, C = Vh, Bi = dgi and Di = dli, i = I, m, 

according to [l| Proposition 17.10, Theorem 18.15], D^^ = \7l* is a monotone and z/j- 
Lipschitzian operator for i = 1, ...,m. The monotone inclusion problem (18) reads 



findxGn such that z € df{x) + ^ L*{{dgiUdli){LiX - r^)) + Vh{x), 



(38) 
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while the dual inclusion problem (19) reads 
find .1 G e such that 3x G ^ : | _ ^ (4n4)(i.^ " r.),^ = 1, 

(39) 

If (x, vi, fm) G "Hx C/i X ... X Qm is a primal-dual solution to (38)-(39), namely, 

m 

z — ^ L*tJj G + '\7h{x) and G {dgindli){LiX — ri), i = 1, m, (40) 



i=l 



then X is an optimal solution of the problem (36), {vi, ...,Vm) is an optimal solution of 
(37) and the optimal objective values of the two problems coincide. Notice that (40) is 
nothing else than the system of optimality condition for the primal-dual pair of convex 
optimization problems (36)-(37). 

The assumptions made on /j guarantees that giOli G ^{Gi) (see [l| Corollary 11.16, 
Proposition 12.14]) and, since dom{gi[Jli) = domgi +domZi,i = l,...,m, one can can 
consider the following qualification condition of interiority-type in order to guarantee ( 40 ) 

(r-i, ...,rm) G sqri JJ(dom5rj + domk) - {{Lix, ...,Lmx) : x G dom/} . 



(41) 



\i=l 



The following two statements are particular instances of Algorithm 10 and Theorem 



11 respectively. 



Algorithm 19 

Initialization; Choose fi > such that 

^ < min {tV^', Sl/i^l ...,5l/yl, Vt/ (E,-=i \\L^\\V5^)}, 
T = /u/(27), ai = /i/(2(5j), i = 1, ..,m, 

e G [2/(2 + ^), 1] and (xq, ui,o, ^^m,o) G "^x Qi X---X Qm- 
For n > set; Xn+i = prox^j [xn - t{YjT=i ^'^^,n + Vh{xn) - z)] 

Un = Xn+1 + 0{Xn+l - Xn) 

Vi^n+1 = prox^^^* [t!i,„ + ai{Liyn - Vl*{vi^n) - ri)], i = 1, ...,m. 

Theorem 20 Suppose that f + h is ^-strongly convex for j > 0, g* + I* is 6i-strongly 
convex for 5i > 0, i = l,...,m, and the qualification condition (41) holds. Then there 
exists a unique optimal solution x to (36), a unique optimal solution (vi, ...,Vfn) to (37) 
fulfilling the optimality conditions (40) and such that the optimal objective values of the 
problems (36) and (37) coincide. The sequences generated by Algorithm 19 fulfill for any 
n> 



i+i - x|p + (1 - uj)'^6i\\vi 



i=l 



UJ ^\\xi — x\ 



- Vi\ 



+ - xoll^ + /iw^(Li(xi - xq), Vi,o 



i=l 



1=1 



where < w 



2(1+0) 
4+At 



< 1. 
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4 Numerical experiments 



In this section we illustrate the applicability of the theoretical results in the context of two 
numerical experiments in image processing and support vector machines classification. 

4.1 Image processing 



In this subsection we compare the numerical performances of Algorithm 14 with the ones 
in the iterative scheme in Theorem [2] for an image denoising problem. To this end we treat 
the nonsmooth regularized convex optimization problem 

inf \l\\x-bf + XiTV{x) + X2\\Wx\\i\, (42) 

where TV : R'^ — )• M denotes a discrete isotropic total variation, W : M.^ ^ M.^ a the 
discrete Haar wavelet transform with four levels, Ai, A2 > the regularization parameters 
and 5 G M'^ the observed noisy image. Notice that we consider images of size k = M x N 
as vectors x S M*^, where each pixel denoted by Xij, 1 < i < M, 1 < j < N , ranges in the 
closed interval from (pure black) to 1 (pure white). 

Two popular choices for the discrete total variation functional are the isotropic total 
variation TV,. ■ ^ ™ 



' ISO 



M-1 N-1 



i=\ 3=\ 
M-1 Af-1 

+ ^ \Xi+l^N — Xi^N\ + ^ ^MJ+l ~ XM,j\ , 
i=l j=l 



Mk 



and the anisotropic total variation TVg 

M-l N-1 

Xi+l,j Xij\ + Ixjj'-i-i Xij\ 

1=1 j=l 
M-1 N-1 

+ ^ \Xi+l^N — Xi^N\ + y^ \xMj+l — XM,j \ , 
i=l j=l 

where in both cases reflexive (Neumann) boundary conditions are assumed. Obviously, in 



both situations the qualification condition in Theorem 15 is fulfilled. 

Denote 3^ = M*^ x M'^ and define the linear operator L : M*^ — )• 3^, Xij 1— )• {LiXij, L2Xij), 
where 

r ^. . _ / ^i+lJ - ^iJ' if i < M _ r Xij+i - Xij, ifj<N 

The operator L represents a discretization of the gradient in horizontal and vertical direc- 
tion. One can easily check that < 8 and that its adjoint L* : 3^ — t- is as easy to 
implement as the operator itself (cf. [10|). Moreover, since W is an orthogonal wavelet, it 
holds IIW^II = 1. 
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(a) Noisy image, a = 0.06 



(b) Noisy image, a = 0.12 




(c) Denoised image, Ai = 0.035, 
A2 = 0.01, isotropic TV 




(d) Denoised image, Ai = 0.07, 
A2 = 0.01, isotropic TV 



Figure 1: The noisy images in (a) and (b) were obtained after adding white Gaussian noise 
with standard deviation a = 0.06 and a = 0.12, respectively, to the original 256 x 256 
lichtenstein test image. The outputs of Algorithm [14] after 100 iterations when solving 
(42) with isotropic total variation are shown in (c) and (d), respectively. 



When considering the isotropic total variation, the problem (42) can be formulated as 

\ni{f{x) + gi{Lx)+g2{Wx)], (43) 

where / : — M, f{x) = ^ ||x — 6|p + 5[o,i]fc [x) is 1-strongly convex, (/i : 3^ — M is defined 
as5i(n,t;) = \i\\{u,v)\U, where ||(-,-)||x : 3^ ^ M, \\{u,v)\U = T^ZiT^U + <r 

is a norm on the Hilbert space y and g2 : ^ M.,g2{x) = A2||x||i. Take {p,q) E y and 
a > 0. We have 

prox^/(p) = ^[0,1]'= ((1 + cr)'Hp + crb)) . 
Moreover, glip,q) = Ssip,q) and 



Prox^g* {p, q) = Ps {p, q) , 



where (cf. (6] 



S 



{P, q) ^y ■■ ,max ^ + qf j < A 



l<j<M 

i<i<-'v 
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e = 








isotropic TV 


anisotropic TV 






o- = 0.06 o- = 0.12 


a = 0.06 a = 0.12 


Al^ 


jorithm 


in 


21 


373 329 


383 388 


Al^ 


jorithm 


14 




95 180 


126 255 



14 



when 



Table 1: Comparison of the algorithm in |21j and its modification, Algorithm 
denoising the images in Figure 1 (a) and (b) for isotropic TV and anisotropic TV. The 
entries represent the number of iterations needed for attaining a root mean squared error 
for the primal iterates below the tolerance level of e = 10~^. 
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Figure 2: RMSE curves for image denoising: the case of white Gaussian noise with stan- 
dard deviation a = 0.06 



and the projection operator Ps : y ^ S \s defined via 

{Pi,j,qid) ^ Ai (P^-i' g^i) 1 < i < M, 1 < i < A^. 

max|Ai,^p2j. + ^2^1 

Finally, gl{p) = 5^_^,^^^,^^k{p), hence 



On the other hand, when considering the anisotropic total variation, the problem (42) 
can be formulated as 

\ni{f{x) + ~gi{Lx)+g2{Wx)], (44) 

where the functions /, 52 are taken as above and gi : 3^ — )■ M is defined as gi{u,v) = 
Xi\\{u,v)\\i. For every {p,q) e y we have gl{p) = 6^_^_^^^^^k^^_^^^^^^k{p,q) and 

prox,g.(p,g) = Pi^x^Mfxi-x^MfiP^^)- 

We experimented with the 256 x 256 lichtenstein test image to which we added white 
Gaussian noise with standard deviation a = 0.06 and a = 0.12, respectively. We solved 
(42) by Algorithm 14 (with the modifications mentioned in Remark 17) for both instances 
of the discrete total variation functional. For the first noisy image (added noise with 
standard deviation a = 0.06), we took as regularization parameters Ai = 0.035 and 
A2 = 0.01 and for the second one (added noise with standard deviation a = 0.12), Ai = 0.07 
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Figure 3: RMSE curves for image denoising: the case of white Gaussian noise with stan- 
dard deviation a = 0.12 



and A2 = 0.01. As initial choices in Algorithm 14 we opted for A = 1, tq = 50, crifi = 0.0241 
and C72.0 = 0.008. The reconstructed images after 100 iterations for isotropic total variation 
are shown in Figure 1. 



We compared Algorithm 14 from the point of view of the number of iterations needed 
for a good recovery with the iterative method from Theorem [2| (se e [21] ) for both isotropic 
and anisotropic total variation. For the iterative scheme from [21| we have chosen for 
both noisy images and both discrete total variation functionals the optimal initializations 
r = 0.35, cJi = 0.2 and cj2 = 0.01. 

The comparisons concerning the number of iterations needed for a good recovery were 
made via the root mean squared error (RMSE). We refer the reader to Table 1 for the 



achieved results, which show that Algorithm 14 outperforms the iterative scheme in (211. 



Figures 2 and 3 show the evolution of the RMSE curves when solving (43) and (44) with 



the algorithm from 21 and with its modified version, Algorithm 14, respectively. 



4.2 Support vector machines classifications 

The numerical experiments we present in this subsection refer to the class of kernel based 
learning methods and address the problem of classifying images via support vector ma- 
chines. 

We make use of a data set of 11800 training images and 1983 test images of size 
28 X 28 from the website http://www.es. nyu.edu/^roweis/data.html. The problem consists 
in determining a decision function based on a pool of handwritten digits showing either 
the number eight or the number nine, labeled by —1 and +1, respectively (see Figure 
4). We evaluate the quality of the decision function on a test data set by computing the 
percentage of misclassified images. Notice that we use only a half of the available images 
from the training data set, in order to reduce the computational effort. 
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Figure 4: A sample of images belonging to the classes —1 and +1, respectively. 
The classifier functional / is assumed to be an element of the Reproducing Kernel 
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Hilbert Space (RKHS) Ti^, which in our case is induced by the symmetric and finitely 
positive definite Gaussian kernel function 



X M"* M, K{x,y) = exp 



X ■ 



Let (•, be the inner product on T-L^, \\ ■ \\n the corresponding norm and K G M"^*^ the 
Gram matrix with respect to the training data set Z = {(Xi, Yi), y„)} C R'^ x 
{+1, —1}, namely the symmetric and positive definite matrix with entries Kij = K{Xi,Xj) 
for i,j = We make use of the hinge loss function : M x M — M.,v{x,y) = 

max{l — xy,0}, which penalizes the deviation between the predicted value f{x) and the 
true value y E {+1,-1}. The smoothness of the decision function / e Ti^ is employed 
by means of the smoothness functional Q, : — )■ M, 0,{f) = \\f\\f^, taking high values 
for non-smooth functions and low values for smooth ones. The decision function is the 
optimal solution of the Tikhonov regularization problem 



^inf^ I hl{f) + cj2 <fiXi),Yi) I 



(45) 



where C > denotes the regularization parameter controlling the tradeoff between the 
loss function and the smoothness functional. 

By taking into account the representer theorem (see [iS]), there exists a vector c = 
(ci, Cn)^ € such that the minimizer / of (45 ) can be expressed as a kernel expansion 



in terms of the training data, namely, it holds /(•) = ^11=1 ^ii^i.' ^ -^t) ■ ^^i^ case the 
smoothness functional becomes 0(/) = ||/||^ = (/,/)«; = CiCjK(Xj, Xj) = 

c^Kc and for i = 1, ...,n it holds f{Xi) = X]j=i CjK{Xi, Xj) = [Kc)i. This means that in 
order to find the decision function it is enough to solve the convex optimization problem 

\n±\h{c) + Y,9i{Kc)\, (46) 



where /i : M" ^ R, h{c) = \c^Kc is convex, differentiable with ||i^||-Lipschitz gradient 
and Qi : M" — )• M, gi{c) = Cv{ci,Yi), i = 1, ...,n, are convex functions. The optimization 



problem (46) has the structure of (30), where / is taken to be identically 0. Notice that 
in this case the function f + h = h is 7-strongly convex with 7 = Amm, where Amin is the 
smallest eigenvalue of the matrix K. Due to the continuity of the functions gi,i = 1, n. 



the qualification condition required in Theorem 15 is guaranteed. We solved (46) by 



Algorithm 14 and used for fi > the following formula (see [8]) 

prox^g. (c) = (0, . . . , Py^ [.CO] (q - /iFi) , . . . , 0) ^ . 

With respect to the considered data set, we denote by P= {{Xi,Yi),i = 1,...,5899} C 
]^784 ^ {+1, —1} the set of available training data consisting of 2974 images in the class 
— 1 and 2925 images in the class +1. A sample from each class of images is shown in 
Figure 4. The images have been vectorized and normalized by dividing each of them by 



1 

the quantity Ur'^^" 
As initial choices in Algorithm 
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we took To = 0.99||^,A = ||K|| + 1 and 



Y^l + To (27 — ||K||ro)/A/(nro||Kp), i = 1, ...,n, and tested different combinations of the 
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a 


0.15 


0.175 


0.2 


0.25 


0.5 


training 


error 








0.14 


4.09 


49.55 


test 


error 


1.36 


2.12 


3.33 


5.60 


49.12 



Table 2: Misclassification rate in percentage for different choices of the kernel parameter 
a and for both the training and the test data set. 



kernel parameter a over a fixed number of 1500 iterations. In Table 2 we present the 
misclassification rate in percentage for the training and for the test data (the error for the 
training data is less than the one for the test data). One can notice that for certain choices 
of fj the misclassification rate outperforms the one reported in the literature dealing with 
numerical methods for support vector classification. Let us mention that the numerical 
results are given for the case C = 1. We tested also other choices for C, however we did 
not observe great impact on the results. 
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